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Purpose: To compare the internal computer-based scoring with human-based video scoring of 
cataract modules in the Eyesi virtual reality intraocular surgical simulator, a comparative case 
series was conducted at the Department of Clinical Sciences - Ophthalmology, Lund University, 
Skane University Hospital, Malmo, Sweden. 

Methods: Seven cataract surgeons and 17 medical students performed one video-recorded trial 
with each of the capsulorhexis, hydromaneuvers, and phacoemulsification divide-and-conquer 
modules. For each module, the simulator calculated an overall score for the performance rang- 
ing from 0 to 100. Two experienced masked cataract surgeons analyzed each video using the 
Objective Structured Assessment of Cataract Surgical Skill (OSACSS) for individual models 
and modified Objective Structured Assessment of Surgical Skills (OSATS) for all three modules 
together. The average of the two assessors' scores for each tool was used as the video-based per- 
formance score. The ability to discriminate surgeons from naive individuals using the simulator 
score and the video score, respectively, was compared using receiver operating characteristic 
(ROC) curves. 

Results: The ROC areas for simulator score did not differ from 0.5 (random) for hydromaneuvers 
and phacoemulsification modules, yielding unacceptably poor discrimination. OSACSS video 
scores all showed good ROC areas significantly different from 0.5. The OSACSS video score 
was also superior compared to the simulator score for the phacoemulsification procedure: ROC 
area 0.945 vs 0.664 for simulator score (P = 0.010). Corresponding values for capsulorhexis 
were 0.887 vs 0.761 (P = 0.056) and for hydromaneuvers 0.817 vs 0.571 (P = 0.052) for the 
video scores and simulator scores, respectively. The ROC area for the combined procedure was 
0.938 for OSATS video score and 0.799 for simulator score (P=0.072). 
Conclusion: Video-based scoring of the phacoemulsification procedure was superior to the 
innate simulator scoring system in distinguishing cataract surgical skills. Simulator scoring 
rendered unacceptably poor discrimination for both the hydromaneuvers and the phacoemulsi- 
fication divide-and-conquer module. Our results indicate a potential for improvement in Eyesi 
internal computer-based scoring. 
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Background 

Training with surgical simulation is becoming an important part of resident surgi- 
cal training in ophthalmology. 1 A shift from counting cases to competence-based 
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curricula for learning cataract surgery is on its way, and the 
implementation of structured surgical curricula has also been 
shown to have a favorable impact on complication rates. 2 ' 3 In 
many departments, simulation training is a required part of 
the training curriculum. Often, the simulator is also used for 
assessment, and residents are quantitatively evaluated on the 
surgical simulator as a form of diagnostic tool for cataract 
surgical skills. 1 Currently there are two commercially avail- 
able virtual reality surgical simulators for cataract surgery: 
Phaco Vision (Melerit Medical, Linkoping, Sweden) and 
Eyesi (VRmagic, Mannheim, Germany). During the last 
5 years, the Eyesi intraocular surgical simulator has been 
acquired by several institutions around the world. It was the 
predominant simulator used according to a recent survey 
regarding the role of simulators in ophthalmic residency train- 
ing in the US. 1 The Eyesi simulator has been evaluated for 
construct validity. 4,5 It has also been shown that training with 
the simulator improves wet-lab capsulorhexis performance. 6 
One report also suggests an associated improvement in surgi- 
cal performance on real cataract operations. 7 However, using 
the simulator for feasible assessment demands adequate scor- 
ing that allows the distinguishing of cataract surgical skill. 
This study therefore investigates the Eyesi as a diagnostic 
tool for cataract surgical skill by comparing the ability to dis- 
tinguish cataract surgeons from nonsurgeons using either the 
internal Eyesi computer-based scoring or human video-based 
scoring of cataract modules in the Eyesi surgical simulator. To 
our knowledge, this is the first report evaluating a simulator 
with regard to performance score using receiver operating 
characteristic (ROC) curve analysis. 

Materials and methods 

Seventeen medical students and seven cataract surgeons were 
recruited to the study. None of them had prior experience with 



the Eyesi surgical simulator. The students were attending 
their seventh semester, and the simulator training took place 
during their ophthalmology rotation. The cataract surgeons 
worked at the university hospital or at a local hospital in the 
region. The group included five very skilled surgeons with 
over a thousand to several thousand cataract operations, and 
two junior surgeons with 20 and 150 self-reported cataract 
operations. 

Simulator 

The simulator investigated in this study was the Eyesi surgical 
simulator (VRmagic software version 2.5). The cataract head 
was used. The simulator consists of a model eye connected 
to a computer. Probes are inserted and cameras inside the 
eye detect the movement of the probes. A virtual image is 
created on the computer. The virtual image is projected 
on two oculars, creating a binocular virtual image of the 
anterior segment. The image is also shown on an observer 
screen. Several manipulating and procedure-specific modules 
are available in the simulator. For this study, three cataract 
modules representing procedure-specific parts of a cataract 
operation were chosen: capsulorhexis, hydromaneuvers, 
and phacoemulsification (phaco) divide and conquer. In the 
capsulorhexis module (Figure 1), the trainee has to inject 
viscoelastics into the anterior chamber, create a flap of the 
anterior capsule with a cystotome, and create a capsulor- 
hexis using a forceps. Here, level four out of ten was judged 
the appropriate level of difficulty most representing a real 
capsulorhexis procedure. In the hydromaneuvers module 
(Figure 1), level one out of four was judged to be the appropri- 
ate level. Here, the trainee has to create a visible fluid wave 
in the cortex and thereafter rotate the nucleus, showing that 
the hydrodissection is appropriate. For the phaco divide - 
and-conquer module (Figure 1), via the divide-and-conquer 




Figure I In the capsulorhexis procedure, the trainee has to inject viscoelastics into the anterior chamber, create a flap with a cystotome in the anterior capsule, and finally 
complete a circular capsulorhexis. In the hydromaneuvers module, the trainee places a cannula under the rhexis edge and injects liquid solution at the right speed, creating 
a visible fluid wave in the cortex. Afterwards, the trainee rotates the nucleus, proving that an appropriate hydrodissection has occurred. During the phacoemulsification 
(phaco) divide and conquer, the trainee has to create grooves in the nucleus, crack the nucleus into quadrants, and finally consume the quadrants with ultrasonic energy 
using phacoemulsification. 
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technique, the trainee has to create grooves in the nucleus, 
crack the nucleus into four quadrants, and finally remove each 
quadrant with phaco. Level five out of six was used. 

A performance score ranging from 0 to 1 00 points is cal- 
culated by the simulator. The score is based on the efficiency 
of the procedure, target achievement, instrument handling, 
and tissue treatment. 

Each participant performed three trials with each of the 
capsulorhexis and hydromaneuvers modules followed by 
two trials with the phaco divide-and-conquer module, in that 
order. For the purpose of this study, only the second trial was 
used for analysis and video recorded. 

Video evaluation 

The training session as seen on the observer screen was 
video-recorded. The films were randomized and evaluated 
by two experienced cataract surgeons who were blinded to 
the identity of the trainees. For evaluation of the saved video 
films, the Objective Structured Assessment of Cataract Surgi- 
cal Skill (OSACSS) tool was used. 8 As previously described, 
this tool has been used to evaluate real cataract operations as 
well as video recordings from simulator training sessions. 9 
The films were also evaluated with the modified Objective 
Structured Assessment of Surgical Skills (OSATS) tool, 
which has also been used for evaluation of real operations 
and simulator films. 10 ~ 12 The evaluations render each proce- 
dure a video-evaluation score where a higher score signifies 
higher surgical skills. Using OSACSS for capsulorhexis, 
the score lies between 3 and 15, for the hydromaneuvers, 
the score lies between 1 and 5, and for phaco divide and 
conquer, the score lies between 5 and 25. Using OSATS, 
all three procedures - capsulorhexis, hydromaneuvers, and 
phaco divide and conquer - are evaluated together, rendering 
scores that range between 4 and 20. 

Statistical analysis 

ROC curves were calculated for the simulator score and for 
the OSACSS and OSATS scores. The ROC curves for the 
OSACSS and OSATS scores were compared with the internal 
computer-based scoring, and the difference was evaluated 
using an algorithm suggested by DeLong et al. 13 Interrater- 
reliability scoring of the OSACSS and OSATS, respectively 
was analyzed using intraclass correlation coefficients. 

Results 

ROC-curve areas and the corresponding ROC curves are 
shown in Figure 2. The ROC area did not differ from 0.5 
(random guess) for the simulator score for the modules 



hydromaneuvers and phaco divide and conquer. The ROC 
areas for video evaluations differed from 0.5 on all modules. 
See Figure 2 for P-values. 

ROC curve-area values were higher for the video evalu- 
ations than for the simulator score, significantly so for the 
phaco divide-and-conquer module (P= 0.01). The differences 
for the other modules were not significant (capsulorhexis 
P = 0.056, hydromaneuvers P = 0.052). 

Calculating the ROC curve for the combined procedure of 
capsulorhexis, hydromaneuvers, and phaco divide and con- 
quer, where the scores from the individual procedures were 
added, yielded an ROC-curve area of 0.799 for the simulator 
scoring and 0.938 for the video-based scoring by OSATS. 
This difference was nonsignificant (P = 0.072). 

Interrater correlation coefficients were high for the 
OSACSS capsulorhexis (r = 0.788), OSACSS phaco 
divide and conquer (r = 0.726), and OSATS (r = 0.764), 
and moderate for the OSACSS hydromaneuvers module 
(r= 0.598). 

Discussion 

The introduction of eye-surgery simulators for cataract- 
surgery training has been beneficial. Previous studies have 
shown construct validity for the Eyesi surgical simulator in 
the anterior segment. 4 - 512 Our results further support these 
findings. 

It is advantageous that training can take place in a safe 
and standardized environment. Surgical simulators provide 
these conditions. Evaluation and assessment is often used 
in simulation-based training, where the trainee has to meet 
preset target criteria. 14 ~ 16 We decided to test the simulator as 
a diagnostic tool for cataract surgical skill by investigating 
the ROC curves. We found that the simulator scoring was 
weak on the hydromaneuvers and phaco divide-and-conquer 
modules, and comparing the ROC area for simulator score 
with 0.5, ie, random guess or flipping coins, rendered unac- 
ceptably poor discrimination for both modules. In contrast, 
the ROC areas using video-based scoring showed high values, 
indicating good diagnostic performance. One problem, espe- 
cially with the phaco divide-and-conquer module scoring, 
was that the window of success was small. Therefore, even 
a small mistake could render a zero score. On the other hand, 
instruments could also be handled inappropriately with little 
effect on the score. 

A limitation in our material is that our group of cataract 
surgeons was quite small, and represents a large variation in 
cataract surgical skills. Removing the two junior surgeons 
improved the ROC curves for both simulator scoring and 



Clinical Ophthalmology 2013:7 



submit your manuscript 



1975 



Dovepress 



Selvander and Asman 



Dovepress 




1.0 



0.8 



~ 0.6 
> 

'tfl 

0) 0.4 
CO 



0.2 



0.0 



Phaco divide and conquer 





/ AUC =0.945 








P = 0.001* 








auc 


=0.664 






P = 


0.216 












Video 








i 


Simulator 



0.0 0.2 0.4 0.6 0.8 

1 -specificity 



1.0 



Combined procedures 



1.0 



0.8 



0.6 

> 



tfl 
c 

0) 0.4 
CO 



0.2 



0.0 



AUC =0.938 

p= ojoor 



AUC =0.799 
P = 0.025* 



■ Video 

■ Simulator 



0.0 



0.2 



0.4 0.6 

1 -specificity 



0.8 



1.0 



Figure 2 Receiver operating-characteristic curves representing discrimination of cataract surgeons from naive individuals for capsulorhexis, hydromaneuvers, and 
phacoemulsification (phaco) divide-and-conquer modules, as well as combined procedures. Area under the curve (AUC) values are shown for simulator score and video 
score (Objective Structured Assessment of Cataract Surgical Skill for capsulorhexis, hydromaneuvers, phaco divide and conquer, Objective Structured Assessment of Surgical 
Skills for combined procedure), with respective P-values representing difference from hazard area (0.5). *Statistical significance. 



video scoring for capsulorhexis (ROC areas 0.835 and 0.965, 
respectively) and simulator scoring for hydromaneuvers 
(ROC area 0.581). On the other hand, it did not improve the 
ROC curve for the phaco-module simulator scoring (ROC 
area 0.653), further suggesting that the scoring needs to be 
developed further. 

Our study was done with study participants' first 
encounter with the simulator. One could argue that with a 
few more training iterations, the discrimination between 
surgeons and naive operators would be easier. However, for 
an ideal simulator, surgeons should score well even without 
more extensive simulator practice. Furthermore, since the 



surgeons' activities yielded good discrimination based on 
the videos, the problem seems to be related to the scoring 
levels rather than the simulator as such. Another issue with 
the innate scoring is the truncation of "poor performance," 
rendering zero points at sometimes minor mistakes. Also, 
the innate simulator score measures how well the goal was 
reached, how long the performance took, and the number of 
actual adverse events. It seems that the human-based scor- 
ing also captured potentially hazardous situations that the 
trainee encountered, besides actual performance. It would 
be beneficial if the simulator-scoring protocol would also 
address these cognitive processes. 
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The ability to distinguish cataract surgeons from naive 
individuals shown in this study supports the construct 
validity of the Eyesi surgical simulator. Cataract surgeons 
in training can train with the simulator, and with training 
acquire skills that experienced surgeons have. It is known 
that complication rates, such as for posterior capsular 
rupture, are much higher for new cataract surgeons compared 
to experienced ones, and that it takes about 160 operations 
before these learning curves associated with cataract training 
flatten out. 17 Our belief is that the Eyesi intraocular surgery 
simulator is a beneficial training tool for aspiring cataract 
surgeons. For it to be used as an assessment tool based on 
the simulator scoring requires refinement of scoring though. 
Previous studies have shown concurrent validity for the cap- 
sulorhexis procedure, but our suggestion is that, at least for 
the phaco divide-and-conquer and hydromaneuvers modules, 
it is more reliable to evaluate the video-based procedure if 
the simulator is to be used as an assessment tool. 912 Further 
development of the computer-based scoring system, tak- 
ing into account the cognitive processes embedded in the 
human-based evaluation, is likely to be a successful route 
that deserves further research. 
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